Schema-First & Design-First: The Backbone of Our Scalable Backend System

Posted on : 09-08-2025 | 4 min reading time

This article outlines how Schema-First and Design-First principles were applied to build a scalable, maintainable backend for high-frequency vehicle CAN data processing and connected-vehicle services. Using a microservices architecture and predefined sequence flows, we ensured consistency in data handling, clarity in service boundaries, and flexibility for future expansion without major redesign.

1. Schema-First: Our Universal Data Language : We used the Schema-First method to design all our data formats, naming conventions, and validation rules before building any service logic. That meant: A strict Redis key format {Vehicle-Id}_{can_frame_id} used in all telemetry caching. DBC schema definitions for decoding every CAN message into structured fields — used across parsing, storage, and analytics. Standard TTL rules for real-time cache entries so no stale data is served. Predefined API payload structures so services can integrate without guesswork. Because we set these schemas up front, all microservices and APIs operate on the same definitions — whether fetching telemetry, generating reports, or managing users.

2. Design-First: Architecture Before Code : We used the Design-First method to plan the entire system architecture before implementation, mapping how every service and database would interact. We decided: Service boundaries — e.g., only the Python CAN Parser Service decodes raw data, only Redis handles short-term caching. Data flow paths — from ingestion → parsing → caching → storage → reporting. Batch processing rules — e.g., ClickHouse writes happen in batches of pre defined limit size or on time-based flush. Integration points — clearly defined hand-offs between microservices. This planning meant we could scale, replace, or enhance individual services without breaking the rest of the system. Our reporting service was designed from the start to pull data directly from ClickHouse and Postgres using the pre-defined schemas. Whether generating daily, weekly, monthly, or quarterly reports, all follow the same standardized retrieval patterns.

3. Microservices We Built Using This Approach : Every microservice in our backend was designed using Schema-First and Design-First: API Gateway – Routes and secures requests. Backend Service – Orchestrates logic and API responses. Python CAN Parser Service – Decodes raw CAN data using the DBC schema. Redis Service – Ultra-fast caching layer. ClickHouse Service – Large-scale analytics storage. Postgres Service – Relational and transactional data. Reporting Service – Generates daily, weekly, monthly, quarterly reports. Logging Service – Captures all events for monitoring.

4. Sequence Flows: Enforcing Consistency in All Operations From the sequence diagram, we designed and applied flows for every backend function — from login to vehicle data retrieval, user management, telemetry fetching, and reporting. We always follow the same pattern: Authentication first via API Gateway & Firebase. Payload/body verification to validate structure and required fields. Service-specific logic execution (fetch, insert, update, delete). Database or cache interaction according to predefined paths. Structured response in the agreed schema. This same pattern is applied whether the operation is: Adding a vehicle Fetching battery/motor/fault data etc …… Generating reports Managing users

5. Why This Worked for Everything We Built : By using Schema-First and Design-First methods for the entire project: Consistency – All services speak the same data language. Clarity – No confusion about who owns which function or dataset. Scalability – Adding new features or scaling services doesn’t cause data mismatches. Reliability – Sequence flows keep all operations secure and predictable

Final Takeaway We built our backend on Schema-First and Design-First from day one — every service, API, and data flow follows the same clear rules. That’s why we can add features, connect new tools, and scale without breaking a thing.